Audit of KETO-CTA Study Results

Dataset of Plaque Metrics

Longitudinal Data From the KETO-CTA Study: Plaque Predicts Plaque, ApoB Does Not

Released by organization funding the study:

https://citizensciencefoundation.org/keto-cta/

All code, figures, data, studies available at:

https://github.com/SloughJE/keto_cta_analysis_check

Incorrect Y-Axis Tick Labels

Figure 1B from Study

Figure 1B from Data

Individual Change in Plaque Volume

(B) The red line represents the median change (0.8%), and the shaded area represents the IQR (0.3%-1.7%).

  x     y

q25 1 15.5 q75 2 25.2

q25 1 102. q75 2 163.

Incorrect Shading Area

Figure 1A from Study

Figure 1A from Data

Individual Change in Plaque Volume

(A). The red line represents the median change (18.9 mm3), and the shaded area represents the IQR (9.3-47.0 mm3).

Incorrect Y-Axis Tick Labels

Figure 2F from Study

Figure 2F from Data

Changes in Total Plaque Score vs Coronary Artery Calcium

(C, F) Only CAC is associated with changes in NCPV and TPS. The regression line was fitted with the function “lm,” which regresses y~x, and the shaded area represents the standard error.

Linear Model Assumptions

4 Simple Linear Regression Assumptions

3 are tested with data

Linearity: between the predictor and the outcome
Constant variance (homoscedasticity) of residuals
Normally distributed residuals

These linear assumptions are quantifiable and objectively testable.

If the assumptions don’t hold, statistical significance and uncertainty estimates aren’t trustworthy
Results may be invalid

Violations

“all linear model assumptions were corroborated with the R function performance::check_model.”

- Direct quote from the study

Actual Assumption Tests

Model	β	Linearity	Constant Variance	Residual Normality
ΔNCPV ~ CAC_bl	β = 0.18 p = <0.001	Violation p = 0.031	Violation p = 0.001	Violation p = <0.001
ΔNCPV ~ NCPV_bl	β = 0.25 p = <0.001	OK p = 0.198	Violation p = <0.001	Violation p = <0.001
ΔNCPV ~ PAV_bl	β = 5.48 p = <0.001	Borderline p = 0.050	Violation p = <0.001	Violation p = <0.001
ΔNCPV ~ TPS_bl	β = 7.37 p = <0.001	OK p = 0.132	Violation p = <0.001	Violation p = 0.001

Objective tests show all 4 models failed at least 2 tests.

Response from Authors

“The validity of the regression models is also questionable. Despite claims of meeting assumptions, variables were reported using medians, suggesting non-normal distributions. Visual inspection of scatter plots shows clustering and no clear linear trends. Robust or nonparametric methods might have been more appropriate, and model diagnostics would improve transparency.”

- quote from letter to the editor

“we are aware of the relevance of linear assumptions to obtain accurate estimators. Because residual plot evaluation can also be subjective, we followed their suggestion and reran all models with robust linear regression (using the MASS::rlm function in R version 4.4.3 [R Foundation]). While, as expected, there were small differences with the published estimates, all models using robust regression were consistent with what was reported.”

- quote from the reply to a letter to the editor

“residual plot evaluation can also be subjective”

- quote from the reply to a letter to the editor

“we are aware of the relevance of linear assumptions to obtain accurate estimators. Because residual plot evaluation can also be subjective, we followed their suggestion and reran all models with robust linear regression (using the MASS::rlm function in R version 4.4.3 [R Foundation]). While, as expected, there were small differences with the published estimates, all models using robust regression were consistent with what was reported.”

- quote from the reply to a letter to the editor

Subjective Assumptions

Calling residual plot evaluation “subjective” is misleading and evasive.

Visual checks are interpretive, but these assumptions objectively testable with quantifiable methods.

“reran all models with robust linear regression…MASS::rlm”

Robust Linear Regression mainly just down-weights outliers.

Does not deal with non-linearity, heteroskedasticity and non-normality of residuals.

check_model output:

Conclusions vs Actual Models Reported

“CONCLUSIONS In lean metabolically healthy people on KD, neither total exposure nor changes in baseline levels of ApoB and LDL-C were associated with changes in plaque.”

Abstract claim component	Model	Model reported
Δ-plaque vs ΔLDL-C	Δ-NCPV ~ ΔLDL-C	Not reported
Δ-plaque vs LDL-C exposure	Δ-NCPV ~ LDL-C exposure	Not reported
Δ-plaque vs LDL-C baseline	Δ-NCPV ~ LDL-C baseline	Not reported
Δ-plaque vs ΔApoB	Δ-NCPV ~ ΔApoB	Reported
Δ-plaque vs ApoB exposure	Δ-NCPV ~ ApoB exposure	Not reported
Δ-plaque vs ApoB baseline	Δ-NCPV ~ ΔApoB	Reported
N/A	NCPV_final ~ LDL-C exposure	Reported (NCPV_final, PAV_final)

No TPS Model Results

“Results Neither change in ApoB …, baseline ApoB, nor total LDL-C exposure … were associated with the change in noncalcified plaque volume (NCPV) or TPS.”

“Neither … change in ApoB nor the ApoB level … were associated … with TPS (Figures 2D and 2E, Table 3).” - “changes in and baseline levels of ApoB were not associated with changes in NCPV or TPS”

Figures 2D–2F are Δ-TPS (outcome) panels (vs ΔApoB, ApoB, CAC_bl)

Table 3 has no Δ-TPS models

No Δ-TPS ~ LDL-C exposure models or results anywhere.

Lifetime LDL-C Exposure Calculation

“LDL-C exposure on a KD was calculated by summing the products of the reported days on a KD prior to study commencement and baseline LDL-C on a KD plus the study follow-up days by their final LDL-C.”

\[ \text{LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \]

one baseline value for all pre-study KD time and one final value for all follow-up is a coarse simplification
relies on recall of KD start

standard AUC/time-weighted approach (need multiple measurements)
limitation due to resources

Lifetime LDL-C Exposure Calculation

“LDL-C exposure on a KD was calculated by summing the products of the reported days on a KD prior to study commencement and baseline LDL-C on a KD plus the study follow-up days by their final LDL-C.”

\[ \text{LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \]

“Estimated lifelong LDL-C additionally included the product of age upon commencing a KD and pre-KD LDL-C.”

\[ \text{Life-LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \;+\; \boldsymbol{\big( Age_{\text{at-KD-start}}\cdot LDL_{\text{pre-KD}} \big)} \]

This equation is nonsensical. It is invalid.

It adds days and age together. It’s like adding miles and inches.
pre-KD term is down-scaled by ~365× relative to the day-based terms.
any associations with outcomes are dominated by keto diet exposure.
if they did convert age, “lifetime” exposure just reflects how old someone was at KD

Age “mediation” analysis

“Estimated lifetime LDL-C exposure was only a significant predictor of final NCPV in the univariable analysis but lost significance when age was included as a covariate (Table 3). Both age and lifetime LDL-C exposure lost significance when baseline CAC was included in the model (Table 3).”

A mediation analysis:

tests how an exposure affects an outcome through a middle step (the mediator)
estimates an indirect effect (through the mediator) and a direct effect (everything else).
requires a pre-specified pathway and proper statistical testing (usually with CIs or bootstraps).

This is not a mediation analysis.

No indirect effect was estimated or tested.
Age can’t be a mediator of LDL exposure (age isn’t caused by LDL); it’s a confounder.
They only compared p-values after adding variables.

Exposure (E): lifetime LDL-C exposure Mediator (M): baseline CAC (CAC_bl) Outcome (Y): final NCPV Confounder (A): age (affects E, M, and Y)

The mediated/indirect path you’d test is: E → M → Y (LDL exposure → baseline CAC → final NCPV).
When you include M = baseline CAC in the model, you block that E → M → Y path and estimate only the direct effect of E on Y (the E → Y arrow not going through M).

So “Adding baseline CAC blocks the pathway → tests only the direct effect” means: conditioning on baseline CAC removes the indirect route through CAC, leaving only the unmediated E → Y association to be estimated.

What mediation is (A → C through B):

Specify roles: Exposure (A), Mediator (B), Outcome (C).

Fit two models:

B ~ A (+ confounders) → path a

C ~ A + B (+ confounders) → paths b, c′

Report the indirect effect (a×b) with CIs (usually bootstrapped). State assumptions.

Only shows outcome regressions (NCPV_final ~ Age, then + Life-LDL-C_exp, then + CAC_bl).

No mediator model, no indirect effect, no CIs.

The “mediator” (Life-LDL-C_exp) is partly built from Age and mixes units → mathematical coupling; any drop in the Age coefficient can be mechanical.

Age “mediation” analysis

“Estimated lifetime LDL-C exposure was only a significant predictor of final NCPV in the univariable analysis but lost significance when age was included as a covariate (Table 3). Both age and lifetime LDL-C exposure lost significance when baseline CAC was included in the model (Table 3).”

They ran (reported) three regressions in sequence.

Conclusion: after adjusting for baseline CAC, neither age nor lifetime LDL-C predicts NCPV_final.

CAC explains the association; age / lifetime LDL-exposure don’t matter.

Changes in p-values come from collinearity due to Age being embedded in lifetime LDL-C exposure.

Age is a confounder/proxy for exposure duration, not a mediator.

They did NOT report NCPV_final ~ lifetime LDL-C exposure model results.

Bayes Factor R-scale

“Bayes factors were calculated…with default settings and an ~ rscale value of 0.8 to contrast a moderately informative prior with a conservative distribution width (to allow for potential large effect sizes) due to the well-documented association between ApoB changes and coronary plaque changes.”

Calling 0.8 “moderately informative” is inaccurate.

R package docs: “medium”, “wide”, “ultrawide” = 0.354, 0.5, 0.707

rscaleCont = 0.8 is wider than “ultrawide” → a very diffuse prior that places substantial mass on very large effects
No reported prior-sensitivity to alternative r-scales
Same r-scale apparently used for all models, no justification.

Fixed description:

“We used a very wide prior on coefficients (rscale = 0.8, wider than the package’s”ultrawide”), which places substantial prior mass on very large effects. This diffuse prior penalizes small-to-moderate effects, requiring substantially stronger evidence than under the ‘wide’ or ‘medium’ defaults to support them.”

Conclusions from Univariable Δ-Models

“Linear models on the primary (NCPV) and secondary outcomes were univariable”

All conclusions based on independent univariable change score models

Δ-NCPV ~ ApoB; Δ-NCPV ~ NCPV_baseline

Confounding unchecked (age, sex, BP, CAC_bl, baseline plaque)
Δ-NCPV contains baseline NCPV, biasing associations (mathematical coupling)
less power, more noise
ignores starting plaque and basics like age/sex/blood
baseline imbalance and shared correlates can inflate, reduce, or reverse associations

Consensus in biostatistics: use ANCOVA style model, follow-up ~ baseline + covariates for pre/post analyses.

NCPV_follow-up ~ NCPV_baseline + ApoB_baseline + age + sex + systolic_BP

Conclusions from Univariable Δ-Models

ΔNCPV ~ ApoB_baseline (unadjusted change score)

Question: “Among all participants, is baseline ApoB associated with the raw change in plaque over 1 year, without accounting for where people started or other factors”?

Slope interpretation: Average difference in change (mm³) per 1 mg/dL higher ApoB, unadjusted.

Clinically limited: Doesn’t condition on starting plaque, age, sex, BP; easily distorted by baseline differences and extra noise in change scores.
It doesn’t tell an individual with given baseline/risk profile what their 1-year plaque will be.

NCPV_follow-up ~ NCPV_baseline + ApoB_baseline + age + sex + systolic_BP (ANCOVA)

Question: Among people with the same starting plaque, age, sex, and BP, do those with higher ApoB tend to have more plaque at 1 year?

Slope interpretation: Expected difference in 1-year NCPV (mm³) per 1 mg/dL higher ApoB, holding baseline NCPV and covariates fixed.

Clinically relevant: Aligns with what a patient asks: “Given where I start and my risk factors, does higher ApoB mean more plaque next year, and by how much?”

Optionally test ApoB × NCPV_baseline to allow baseline-dependent progression.

Preprint vs Published Version

Pre-print title: Plaque Begets Plaque, ApoB Does Not; Longitudinal Data From the KETO-CTA Trial

Published title: Longitudinal Data From the KETO-CTA Study; Plaque Predicts Plaque, ApoB Does Not

In text: find and replace ‘begets’ with ‘predicts’.

Description of main figure, and primary outcome:

“Most participants presented with stable NCPV (Figures 1A and 1B), with 1 participant exhibiting a decrease in NCPV (Figures 2A to 2C) and 6 participants showing decreases in TPS scores over 1 year”

Removed and replaced with:

“The median change in NCPV was 18.9 mm3 (IQR: 9.3-47.0 mm3) and the median change in PAV was 0.8% (IQR: 0.3%-1.7%).”

In CONCLUSIONS section: “In an exploratory analysis, changes in and baseline levels of ApoB were not associated with changes in NCPV or TPS…”

“exploratory analysis” added in published version. Only mention in entire paper.

Table 1 median (Q1–Q3) PAV at baseline changed from 1.25% (0.5–3.6) to 1.6% (0.5–4.9).

Record of pre-print is unavailable. All links go to new version. It is available here.